In a computing environment using distributed or shared storage, computer storage may be provided to one or more users or applications using a highly abstracted infrastructure. This means that the characteristics and locations of the disk drives, storage arrays, and servers where the actual storage takes place are typically hidden from the user or application accessing the storage. The user or application accesses the distributed storage by referencing its symbolic or virtual location, and the distributed storage system automatically translates the virtual location into a physical location where the requested storage is actually stored and forwards the storage request to the physical device at that location. This allows the vendor providing the storage to exercise extensive flexibility in deciding how and where to implement the storage as the distributed storage system may simply change how it translates the virtual location requested by the user or application. This includes the ability to move storage from one storage device to another to address capacity, workload, and/or other requirements. These changes in implementation details are often hidden or transparent from the application or the user, which access the storage by making storage requests using an interface, such as an application programming interface (API), and providing the virtual location information for the requested storage. These virtualized and/or abstracted features of distributed storage systems may make them useful in cloud computing systems.
And while distributed storage provides great flexibility to the storage provider, it often comes with some cost to the application or user. For example, distributed storage is typically accessed over a network, such as the Internet. This may add significant overhead to storage requests as both the storage request and the response to the storage request may travel across the network. At a minimum this introduces latency or delay in the handling of the storage requests. As the storage system is being used to handle storage requests, one or more of the components and/or systems between the client or host where the storage request originates and the storage device where the storage request is ultimately handled may introduce unexpected delay and/or introduce a bottle neck in the handling of the storage requests. For example, when storage requests are transmitted using a low bandwidth and/or heavily overloaded network connection, this may introduce undesirable, but possibly avoidable, delay in the handling of the request. As another example, when a storage device, such as a disk drive, in the storage system receives a large number of storage requests over a short period of time, the processing of these requests may be undesirably delayed. Given the complexity of the distributed storage system, it is not generally a simple task to determine when, where, and/or why undesirable delays are being introduced into the handling of storage requests. To begin to answer this question, each of the various components and/or systems are typically monitored.
A subsystem often used to reduce the delay in the handling of storage requests is a cache or caching system. Caching systems are based on the general observation that once a storage request is made for a particular storage block, a follow-up request within that same storage block is more likely than not to occur in the near future. To take advantage of this, a higher speed storage device is installed somewhere in the processing path between the client or host where the storage request originates and the storage device where the storage request is ultimately handled. In some distributed storage systems there may be multiple caches or caching systems at different points along the processing path. Memory in the caching system is used to store data associated with recent storage requests as one or more cache blocks so that follow-up requests to the same storage may be more rapidly handled by accessing the cache blocks, rather than involving a generally slower read and/or write operation on a storage device as well as a potentially latency inducing round trip across the network. As storage requests are processed they are typically routed through the caching system. When the caching system receives the storage request, the memory in the caching system is checked before forwarding the storage request for further handling.
In some caching systems, the speculative caching of storage blocks also occurs. Speculative or prefetch caching is based on the further observation that once a storage request is made for a particular storage block, a follow-up request to storage blocks that are after the particular storage block in the virtualized address space for the same storage volume is likely to occur in the near future. For example, when a request is made for the third storage block of a storage volume, there is generally a high likelihood that a request to the fourth storage block of the storage volume may be made in the near future. In some caching systems the prefetch caching may be accommodated using a resource-available approach where prefetching occurs when the other components or subsystems have capacity to handle the prefetch requests.
Caching systems are typically monitored to determine how well they are operating. One common caching metric is hit rate. The hit rate measures the ratio of storage requests that may be satisfied using the caching system to the total number of storage requests. In some caching systems the hit rate may be monitored for individual tenants or users of the storage system such as particular clients and/or storage volumes. For many caching systems, the hit rate may be 90% or higher indicating that 90% or more of the storage requests may be handled without forwarding the storage request to another storage device. A low hit rate, however, indicates little more than the hit rate is low and generally provides no indication as to why it might be low. From the low hit rate alone it is not possible to determine whether the hit rate is below because of a poor cache configuration, an unfortunate sequence of storage requests, and/or contention among multiple tenants in the use of the caching system.
Caching systems employing prefetching may also monitor their activities. This may include monitoring a number of storage blocks prefetched and/or a ratio of the number of storage blocks prefetched relative to the number of storage requests. Both of these metrics, however, provide little to no insight on whether the prefetched storage blocks are improving the hit rate for the caching system or whether they may be consuming storage system resources without improving the hit rate.
Accordingly, it would be desirable to provide improved monitoring of caching systems.
In the figures, elements having the same designations have the same or similar functions.